Imaging via diffuser modulation by translating a sample

ABSTRACT

An imaging system includes a sample mount for holding a sample to be imaged, a light source configured to emit a light beam to be incident on the sample, a translation mechanism coupled to the sample mount and configured to scan the sample to a plurality of sample positions in a plane substantially perpendicular to an optical axis of the imaging system, a mask positioned downstream from the sample along the optical axis, and an image sensor positioned downstream from the mask along the optical axis. The image sensor is configured to acquire a plurality of images as the sample is translated to the plurality of sample positions. Each respective image corresponds to a respective sample position. The imaging system further includes a processor configured to process the plurality of images to recover a complex profile of the sample based on positional shifts extracted from the plurality of images.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a non-provisional application of and claims thebenefit and priority under 35 U.S.C. 119(e) of U.S. ProvisionalApplication No. 62/825,120, filed Mar. 28, 2019 entitled“SUPER-RESOLUTION IMAGING VIA TRANSLATED PATTERN ILLUMINATION ANDTRANSLATED PATTERN MODULATION,” and U.S. Provisional Application No.62/832,403, filed Apr. 11, 2019, entitled “SUPER-RESOLUTION IMAGING VIATRANSLATED PATTERN ILLUMINATION AND TRANSLATED PATTERN MODULATION,” theentire contents of which are incorporated herein by reference for allpurposes.

STATEMENT AS TO RIGHTS TO INVENTIONS MADE UNDER FEDERALLY SPONSOREDRESEARCH AND DEVELOPMENT

This invention was made with government support under Grant No. 1510077awarded by the National Science Foundation. The government has certainrights in the invention.

The following two U.S. Patent Applications (including this one) arebeing filed concurrently, and the entire disclosure of the otherapplication is incorporated by reference into this application for allpurposes:

Application Ser. No. 16/819,042, filed Mar. 14, 2020, entitled “IMAGINGVIA TRANSLATED SPECKLE ILLUMINATION AND TRANSLATED DIFFUSER MODULATION”,and

Application Ser. No. 16/819,041, filed Mar. 14, 2020, entitled “IMAGINGVIA DIFFUSER MODULATION BY TRANSLATING A SAMPLE”.

TECHNICAL FIELD

Embodiments of the present invention relate to imaging systems, moreparticularly to super-resolution microscopy imaging systems.

BACKGROUND

It may be desirable to achieve a high spatial resolution and a widefield of view (FOV) simultaneously in a microscopy imaging system. Inconventional microscope systems, a combination of an objective lens anda tube lens may be used to image an object. Designing a high numericalaperture (NA) lens with diffraction-limited performance over a largefield of view may be challenging. In addition, conventional microscopesystems with optical lenses tend to be bulky and expensive. Therefore,there is a need for improved microscopy imaging systems.

SUMMARY

According to some embodiments, an imaging system includes a sample mountfor holding a sample to be imaged, a light source configured to emit alight beam to be incident on the sample, a translation mechanism coupledto the sample mount and configured to scan the sample to a plurality ofsample positions in a plane substantially perpendicular to an opticalaxis of the imaging system, a mask positioned downstream from the samplealong the optical axis, and an image sensor positioned downstream fromthe mask along the optical axis. The image sensor is configured toacquire a plurality of images as the sample is translated to theplurality of sample positions. Each respective image corresponds to arespective sample position. The imaging system further includes aprocessor configured to process the plurality of images to recover acomplex profile of the sample based on positional shifts extracted fromthe plurality of images.

According to some embodiments, an imaging system includes a sample mountfor holding a sample to be imaged, a light source configured to emit alight beam to be incident on the sample, a translation mechanism coupledto the sample mount and configured to scan the sample to a plurality ofsample positions in a plane substantially perpendicular to an opticalaxis of the imaging system, and an image sensor positioned downstreamfrom the phase mask along the optical axis. A top surface of the imagesensor is tilted with respect to a surface of the sample. The imagesensor is configured to acquire a plurality of images as the sample istranslated to the plurality of sample positions. Each respective imagecorresponds to a respective sample position. The imaging system furtherincludes a processor configured to process the plurality of images torecover a complex profile of the sample based positional shiftsextracted from the plurality of images.

According to some embodiments, an imaging system includes a sample mountfor holding a sample to be imaged, a light source configured to emit alight beam, the light beam including light in a plurality ofwavelengths, and a light dispersing element configured to disperse thelight beam into a plurality of sub light beams to be incident on thesample at a plurality of angles of incidence. Each respective sub lightbeam corresponds to a respective wavelength and is incident on thesample at a respective angle of incidence. The imaging system furtherincludes a translation mechanism coupled to the sample mount andconfigured to scan the sample to a plurality of sample positions in aplane substantially perpendicular to an optical axis of the imagingsystem, a mask positioned downstream from the sample along the opticalaxis, and an image sensor positioned downstream from the mask along theoptical axis. The image sensor is configured to acquire a plurality ofimages as the sample is translated to the plurality of sample positions.Each respective image corresponds to a respective sample position. Theimaging system further includes a processor configured to process theplurality of images to recover a plurality of complex profiles of thesample based on positional shifts extracted from the plurality ofimages. Each respective complex profile of the sample corresponds to arespective wavelength.

According to some embodiments, an imaging system includes a sample mountfor holding a sample to be imaged, a light source configured to emit alight beam, a diffuser positioned in front of the light source andconfigured to transform the light beam into a speckle illumination beamcharacterized by a speckle pattern, a mirror configured to receive andreflect the speckle illumination beam toward the sample, a scanningmechanism coupled to the mirror and configured to scan the mirror to aplurality of mirror angles such that the speckle illumination beam isincident on the sample at a plurality of angles of incidence, and animage sensor positioned downstream from the sample along an optical axisof the imaging system. The image sensor is configured to acquire aplurality of images as the mirror is being scanned so that the speckleillumination beam is incident on the sample at the plurality of anglesof incidence. Each respective image corresponds to a respective angle ofincidence. The imaging system further includes a processor configured toprocess the plurality of images to recover a complex profile of thesample based on positional shifts extracted from the plurality ofimages.

According to some embodiments, an imaging system includes a sample mountfor holding a sample to be imaged, a light source configured to emit alight beam to be incident on the sample, a mask positioned downstreamfrom the sample along an optical axis of the imaging system, atranslation mechanism coupled to the mask and configured to scan themask to a plurality of mask positions in a plane substantiallyperpendicular to the optical axis of the imaging system, and an imagesensor positioned downstream from the mask along the optical axis. Theimage sensor is configured to acquire a plurality of images as the maskis scanned to the plurality of mask positions. Each respective imagecorresponds to a respective mask position. The imaging system furtherincludes a processor configured to process the plurality of images torecover a complex profile of the sample based on positional shiftsextracted from the plurality of images.

According to some embodiments, an imaging system includes a sample mountfor holding a sample to be imaged, a light source configured to emit alight beam to be incident on the sample, a first transparent platepositioned downstream from the sample along an optical axis of theimaging system, a scanning mechanism coupled to the first transparentplate and configured to rotate the first transparent plate around afirst axis orthogonal to the optical axis so that the first transparentplate is rotated to a plurality of first angles, a mask positioneddownstream from the first transparent plate along the optical axis, andan image sensor positioned downstream from the mask along the opticalaxis. The image sensor is configured to acquire a plurality of images asthe first transparent plate is scanned to the plurality of first angles.Each respective image corresponds to a respective first angle. Theimaging system further includes a processor configured to process theplurality of images to recover a complex profile of the sample based onpositional shifts extracted from the plurality of images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic diagram of an imaging system 100 according tosome embodiments.

FIGS. 2A-2C and 3A-3B illustrate the performance of the imaging systemshown in FIG. 1 using a quantitative phase target as the sampleaccording to some embodiments.

FIG. 2A shows a captured raw image of a phase target under uniformillumination obtained using the imaging system shown in FIG. 1 accordingto some embodiments.

FIG. 2B shows a captured raw image of the a phase target under speckleillumination obtained using the imaging system shown in FIG. 1 accordingto some embodiments.

FIG. 2C shows a recovered image of the phase target obtained using theimaging system shown in FIG. 1 according to some embodiments.

FIG. 3A shows a recovered phase profile along a circle of the phasetarget in FIG. 2C.

FIG. 3B shows a recovered height profile of the phase target forvisualization.

FIG. 4 shows a full field of view reconstruction of a blood smear samplefrom images obtained using the imaging system shown in FIG. 1 accordingto some embodiments.

FIG. 5 shows a schematic diagram of an imaging system according to someembodiments.

FIG. 6 shows a schematic diagram of an imaging system according to someembodiments.

FIG. 7 shows a schematic diagram of an imaging system according to someembodiments.

FIG. 8 shows a schematic diagram of an imaging system according to someembodiments.

FIG. 9 shows a schematic diagram of an imaging system according to someembodiments.

FIG. 10A shows a schematic diagram of an imaging system according tosome embodiments.

FIG. 10B shows a schematic diagram of an imaging system according tosome embodiments.

FIG. 11 shows a schematic diagram of an imaging system according to someembodiments.

FIG. 12 shows a schematic diagram of an imaging system according to someembodiments.

FIG. 13 shows the recovered images of a thick potato sample obtainedusing an imaging system according to some embodiments.

DETAILED DESCRIPTION

Embodiments of the present invention provide various imaging systems forachieving super-resolution imaging via translated speckle illumination,translated pattern modulation, translated phase modulation, andwavelength-encoded mask modulation. In some embodiments, the imagingsystems may not include any optical lens. Such imaging system arereferred herein as lensless imaging systems. Compared with conventionalmicroscope imaging systems, the imaging systems according to embodimentsof the present invention may be able to achieve high spatial resolutionand large field of view at the same time. The achievable spatialresolution may surpass the diffraction-limited resolution ofconventional microscope imaging systems.

The imaging systems according to embodiments of the present inventionmay have applications in digital pathology, quantitative phase imaging,and the like. In addition, these imaging platforms can be employed invisible light imaging systems, coherent X-ray imaging systems, andelectron imaging systems to increase spatial resolution and providequantitative absorption and object phase contrast.

The imaging systems according to embodiments of the present inventionmay afford numerous advantages. For example, by not including anyoptical lens, the imaging systems may be made to be compact, portable,and cost-effective, and therefore may be suitable for deployment inpoint-of-care settings.

Imaging via Translated Speckle Illumination

FIG. 1 shows a schematic diagram of an imaging system 100 according tosome embodiments. The imaging system 100 includes an image sensor 140. Asample 110 to be imaged may be placed above the image sensor 140. Forexample, the sample 110 may be held by a sample mount (not shown in FIG.1). In some embodiments, the distance D between the sample 110 and theimage sensor 140 may be rather small. For example, the distance D may beless than about 1 mm (e.g., about 500 μm).

The imaging system 100 further includes a light source 120. The lightsource 120 may comprise a laser or a light-emitting diode (LED), and isconfigured to emit a coherent or partially coherent light beam. Thelight beam may be collimated, partially collimated, or uncollimated. Theimaging system 100 further includes a diffuser 150 positioned in frontof the light source 120. The diffuser 150 may include an unknown patternformed thereon. Thus, as the light beam emitted by the light source 120passes through the diffuser 150, the light beam may be transformed intoa speckle illumination beam. The imaging system 100 may further includea mirror 130 configured to receive and reflect the speckle illuminationbeam toward the sample 110.

The imaging system 100 further includes a scanning mechanism (not shownin FIG. 1) coupled to the mirror 130 and configured to scan the mirror130 to a plurality of mirror angles, such that the speckle patternincident on the sample 110 has a plurality of translational shifts. Forexample, the scanning mechanism may comprise a galvo scanner. The mirror130 may be scanned in one dimension or two dimensions. For example, themirror 130 may be scanned in the pitch direction (e.g., around an axisperpendicular to the page, so that the speckle illumination beam isscanned left and right in the page), or in the roll direction (e.g., sothat the speckle illumination beam is scanned in and out of the page),or in both the pitch direction and the roll direction.

The image sensor 140, which is positioned downstream from the sample 110along the optical axis 102 of the imaging system 100, is configured tocapture a plurality of images as the speckle illumination beam isincident on the sample 110 at the plurality of angles of incidence. Eachrespective image corresponds to a respective angle of incidence. Theplurality of images may be processed by a processor (not shown inFIG. 1) to produce a complex profile of the sample 110 based oncross-correlations among the plurality of images.

According to some embodiments, to address the positioning repeatabilityand accuracy issues, the positional shifts of the speckle pattern arerecovered based on the phase correlations among of the plurality ofimages. To bypass the resolution limit set by the pixel size of theimage sensor 140, a sub-sampled ptychographic phase retrieval process isused to recover the complex profile of the sample 110. The complexprofile of the sample 110 may include an intensity image as well as aphase image of the sample 110. The reconstruction process may recoverthe unknown speckle pattern as well.

According to some embodiments, the reconstruction process may includethe following steps.

At S101, initialize the complex object O(x,y) (e.g., the sample) and thespeckle pattern P(x,y).

At S102, estimate the j^(th) translated position of the speckle pattern(x_(j),y_(j)) based on image cross-correlation, or other trackingalgorithms such as mutual information optimization and the like.

At S103, according to the imaging model, generate the j^(th) compleximage's exit wave ψ_(j)(x,y) at the image sensor plane based on thetranslated position (x_(j),y_(j)), O(x,y), and P(x,y):ψ_(j)(x,y)=(O(x,y)·P(x−x _(j) ,y−y _(j)))*PSF _(free)(d)=φ_(j)(x,y)*PSF_(free)(d),where (x_(j),y_(j)) is the j^(th) positional shift of the specklepattern, PSF_(free)(d) is the point spread function (PSF) for free-spacepropagation over a distance d, and ‘*’ stands for convolution operation,and φ_(j)(x,y)=O(x,y)·P(x−x_(j),y−y_(j)).

At S104, at the image sensor plane, use the following equation to updatethe exit wave ψ_(j)(x,y) based on the captured intensity imageI_(j)(x,y):

${\psi_{j}^{\prime}\left( {x,y} \right)} = {{\psi_{j}\left( {x,y} \right)}{\left( \frac{\sqrt{{I_{j}\left( {x,y} \right)}_{\uparrow M}}}{\sqrt{{{\psi_{j}\left( {x,y} \right)}}^{2}*{{ones}\left( {M,M} \right)}_{\downarrow M \uparrow M}}} \right).}}$

In the above equation, the image sizes of ψ_(j)(x,y) and I_(j) (x,y) aredifferent. If I_(j) has a size of 100 by 100 pixels, ψ_(j) will have 300by 300 pixels, with an up-sampling factor M=3. The term‘I_(j)(x,y)_(↑M)’ represents the nearest-neighbor up-sampling of thecaptured image I_(j). In the denominator of the above equation, theterm|ψ_(j)(x,y)|² first convolutes with an average filter (M by Mall-one matrix ones(M, M)). It will be then down-sampling by M-timesfollowed by nearest-neighbor up-sampling of M-times. In someembodiments, other up-sampling factor (e.g., M=4, 5, 6, . . . ) may beused.

At S105, propagate the updated ψ′_(j)(x,y) to the object plane and getφ′_(j)(x_(j),y_(j)). Update the object and P(x,y):

${{O\left( {x,y} \right)} = {{O\left( {x,y} \right)} + \frac{{{conj}\left( {P\left( {{x - x_{j}},{y - y_{j}}} \right)} \right)}\left( {\varphi_{j}^{\prime} - \varphi_{j}} \right)}{{\left( {1 - \alpha_{obj}} \right){{P\left( {{x - x_{j}},{y - y_{j}}} \right)}}^{2}} + {\alpha_{obj}{{P\left( {{x - x_{j}},{y - y_{j}}} \right)}}_{\max}^{2}}}}},\mspace{20mu}{{P\left( {{x - x_{j}},{y - y_{j}}} \right)} = {{P\left( {{x - x_{j}},{y - y_{j}}} \right)} + \frac{{{conj}(O)}\left( {\varphi_{j}^{\prime} - \varphi_{j}} \right)}{{\left( {1 - \alpha_{P}} \right){O}^{2}} + {\alpha_{P}{O}_{\max}^{2}}}}},$where ‘conj’ denotes conjugate, and α_(obj) and α_(P) are algorithm.

At S106, j=j+1 and repeat steps S102-S105.

At S107, repeat steps S102-S106 until the solution converges.

It should be appreciated that the specific steps S101-S107 discussedabove provide a particular reconstruction process according to someembodiments. Other sequences of steps may also be performed according toalternative embodiments. For example, alternative embodiments of thepresent invention may perform the steps outlined above in a differentorder. Moreover, the individual steps S101-S107 may include multiplesub-steps that may be performed in various sequences as appropriate tothe individual step. Furthermore, additional steps may be added orremoved depending on the particular applications. One of ordinary skillin the art would recognize many variations, modifications, andalternatives.

The performance of the imaging system 100 were validated using aresolution target, a phase target, and a biological sample. It wasdemonstrated that accurate, high-quality complex images can be obtainedfrom an image set including as few as 10 images. In some embodiments, a6.4 mm by 4.6 mm field of view (FOV) and a half pitch resolution of 1 μmcan be achieved.

FIGS. 2A-2C and 3A-3B illustrate the performance of the imaging system100 using a quantitative phase target as the sample according to someembodiments. FIG. 2A shows a captured raw image under uniformillumination. FIG. 2B shows a captured raw image under speckleillumination. FIG. 2C shows a recovered image of the phase target. FIG.3A shows a recovered phase profile along a circle of the phase target inFIG. 2C. As illustrated, the recovered phase is in a good agreement withthe ground-truth height of the phase target. FIG. 3B shows a recoveredheight profile of the phase target for visualization.

FIG. 4 shows a full field of view reconstruction of a blood smear sampleaccording to some embodiments. 400 raw images were used in thereconstruction process and the speckle pattern was treated as unknown.The imaging area is 6.4 mm×4.6 mm, which is limited by the size of theimage sensor. The insets (a) show the magnified intensity (a1) and phase(a2) of the highlighted regions (a). The insets (b) show the magnifiedintensity (b1) and phase (b2) of the highlighted regions (b). Thus, itwas demonstrated that the imaging system 100 may achieve both highspatial resolution and wide field of view at the same time, which may beimportant for microscopy applications.

Imaging via Pattern Modulation

FIG. 5 shows a schematic diagram of an imaging system 500 according tosome embodiments. The imaging system 500 includes an image sensor 540. Asample 510 to be imaged may be placed above the image sensor 540. Forexample, the sample 510 may be held by a sample mount (not shown in FIG.5).

The imaging system 500 further includes a light source 520. The lightsource 520 may comprise a laser or a light-emitting diode (LED), and isconfigured to emit a coherent or partially coherent light beam to beincident on the sample 510. The light beam may be collimated, partiallycollimated, or uncollimated. The imaging system 500 may include a mirror530 positioned substantially at a 45 degree angle with respect to thepath of the light beam emitted by the light source 520, so as to foldthe light beam for a more compact configuration. The mirror 530 isoptional.

The imaging system 500 further includes a mask 550 positioned downstreamfrom the sample 510 along an optical axis 502 of the imaging system 500,and above the image sensor 540. The mask 550 may include an unknownpattern formed thereon. Thus, as the light beam is transmitted throughthe sample 510 and the mask 550, a diffused image may be formed at theimage sensor 540.

The imaging system 500 further includes a translation mechanism (notshown in FIG. 5) coupled to the mask 550 and configured to scan the mask550 to a plurality of mask positions in a plane (e.g., the X-Y plane)substantially perpendicular to the optical axis 502 of the imagingsystem 500. According to various embodiments, the mask 550 may betranslated in one dimension (e.g., in the X direction or the Ydirection), or in two dimensions (e.g., in both the X and the Ydirections).

As the mask 550 is scanned, the diffused image formed at the imagesensor 540 may shift accordingly. The image sensor 540 is configured tocapture a plurality of images as the mask 550 is scanned to theplurality of mask positions. Each respective image corresponds to arespective mask position. The plurality of images may be processed by aprocessor (not shown in FIG. 5) to produce a complex profile of thesample 510 based on cross-correlations among the plurality of images.

FIG. 6 shows a schematic diagram of an imaging system 600 according tosome embodiments. The imaging system 600 is similar to the imagingsystem 500 illustrated in FIG. 5, but may further include an objectivelens 660 and a tube lens 670 positioned between the mask 550 and theimage sensor 540, as in a conventional microscope.

According to some embodiments, the reconstruction process may includethe following steps.

At S201, initialize the complex object O(x,y) (e.g., the sample) and thediffuser pattern P(x,y).

At S202, estimate the j^(th) translated position of the diffuser pattern(x_(j),y_(j)) based on image cross-correlation, or other trackingalgorithms such as mutual information optimization and the like.

At S203, according to the imaging model, O(x,y) is propagated fordistance ‘d₁’ to the diffuser planeO _(d1)(x,y)=O(x,y)*PSF _(free)(d ₁)

At S204, generate the j^(th) complex image's exit wave ψ_(j)(x,y) at theimage sensor plane based on the translated diffuser position(x_(j),y_(j)), O(x,y), and P(x,y):ψ_(j)(x,y)=(O _(d1)(x,y)·P(x−x _(j) ,y−y _(j)))*PSF_(free)(d)=φ_(j)(x,y)*PSF _(free)(d),where PSF_(free)(d) is the point spread function (PSF) for free-spacepropagation over a distance d, and ‘*’ stands for convolution operation,and φ_(j)(x,y)=O_(d1)(x,y)·P(x−x_(j),y−y_(j)).

At S205, at the image sensor plane, use the following equation to updatethe exit wave ψ_(j)(x,y) based on the captured intensity imageI_(j)(x,y):

${\psi_{j}^{\prime}\left( {x,y} \right)} = {{\psi_{j}\left( {x,y} \right)}{\left( \frac{\sqrt{{I_{j}\left( {x,y} \right)}_{\uparrow M}}}{\sqrt{{{\psi_{j}\left( {x,y} \right)}}^{2}*{{ones}\left( {M,M} \right)}_{\downarrow M \uparrow M}}} \right).}}$

In the above equation, the image sizes of ψ_(j)(x,y) and I_(j)(x,y) aredifferent. If I_(j) has a size of 100 by 100 pixels, ψ_(j) will have 300by 300 pixels, with an up-sampling factor M=3. The term‘I_(j)(x,y)_(↑M)’ represents the nearest-neighbor up-sampling of thecaptured image I_(j). In the denominator of the above equation, the term|ψ_(j)(x,y)|² first convolutes with an average filter (M by M all-onematrix ones(M,M)). It will be then down-sampling by M-times followed bynearest-neighbor up-sampling of M-times. In some embodiments, otherup-sampling factor (e.g., M=4, 5, 6, . . . ) may be used.

At S206, propagate the updated ψ′_(j)(x,y) to the object plane and getφ′_(j)(x_(j),y_(j)). Update the object O_(d1)(x,y) and P(x,y):

${{O_{d\; 1}\left( {x,y} \right)} = {{O_{d\; 1}\left( {x,y} \right)} + \frac{{{conj}\left( {P\left( {{x - x_{j}},{y - y_{j}}} \right)} \right)}\left( {\varphi_{j}^{\prime} - \varphi_{j}} \right)}{{\left( {1 - \alpha_{obj}} \right){{P\left( {{x - x_{j}},{y - y_{j}}} \right)}}^{2}} + {\alpha_{obj}{{P\left( {{x - x_{j}},{y - y_{j}}} \right)}}_{\max}^{2}}}}},{{P\left( {{x - x_{j}},{y - y_{j}}} \right)} = {{P\left( {{x - x_{j}},{y - y_{j}}} \right)} + \frac{{{conj}\left( O_{d\; 1} \right)}\left( {\varphi_{j}^{\prime} - \varphi_{j}} \right)}{{\left( {1 - \alpha_{P}} \right){O_{d\; 1}}^{2}} + {\alpha_{P}{O_{d\; 1}}_{\max}^{2}}}}},$where ‘conj’ denotes conjugate, and α_(obj) and α_(P) are algorithm.

At S207, j=j+1 and repeat steps S202-S206.

At S208, repeat steps S202-S207 until the solution converges.

At S209, propagate the recovered O_(d1)(x,y) to the object plane.

It should be appreciated that the specific steps S201-S209 discussedabove provide a particular reconstruction process according to someembodiments. Other sequences of steps may also be performed according toalternative embodiments. For example, alternative embodiments of thepresent invention may perform the steps outlined above in a differentorder. Moreover, the individual steps S201-S209 may include multiplesub-steps that may be performed in various sequences as appropriate tothe individual step. Furthermore, additional steps may be added orremoved depending on the particular applications. One of ordinary skillin the art would recognize many variations, modifications, andalternatives.

Imaging via Pattern Modulation Using Rotating Slides

FIG. 7 shows a schematic diagram of an imaging system 700 according tosome embodiments. The imaging system 700 includes an image sensor 740. Asample 710 to be imaged may be placed above the image sensor 740. Forexample, the sample 710 may be held by a sample mount (not shown in FIG.7).

The imaging system 700 further includes a light source 720. The lightsource 720 may comprise a laser or a light-emitting diode (LED), and isconfigured to emit a coherent or partially coherent light beam to beincident on the sample 510. The light beam may be collimated, partiallycollimated, or uncollimated. The imaging system 700 may include a mirror730 positioned substantially at a 45 degree angle with respect to thepath of the light beam emitted by the light source 720, so as to foldthe light beam for a more compact configuration. The mirror 730 isoptional.

The imaging system 700 further includes a mask 750 positioned downstreamfrom the sample 710 along an optical axis 702 of the imaging system 700,and above the image sensor 740. The mask 750 may include an unknownpattern formed thereon. Thus, as the light beam is transmitted throughthe sample 710 and the mask 750, a diffused image may be formed at theimage sensor 740.

The imaging system 700 further includes a first transparent plate 760and a second transparent plate 770 positioned between the sample 710 andthe mask 750. The imaging system 700 may further include a scanningmechanism (not shown in FIG. 7) coupled to the first transparent plate760 and the second transparent plate 770, and configured to rotate thefirst transparent plate 760 in one direction (e.g., around the Y-axis,which is perpendicular to the page), and to rotate the secondtransparent plate 770 in an orthogonal direction (e.g., around theX-axis.) The first transparent plate 760 and the second transparentplate 770 may comprise glass slides or any other transparent dielectricmaterials. In some embodiments, the imaging system 700 may include onlyone transparent plate (e.g., the first transparent plate 760 or thesecond transparent plate 770).

As the first transparent plate 760 and the second transparent plate 770are rotated, the diffused image formed at the image sensor 740 may shiftaccordingly. The image sensor 740 is configured to capture a pluralityof images as the first transparent plate 760 is scanned to a pluralityof first angles and the second transparent plate 770 is scanned to aplurality of second angles. Each respective image corresponds to arespective first angle of the first transparent plate 760 and arespective second angle of the second transparent plate 770. Theplurality of images may be processed by a processor (not shown in FIG.7) to produce a complex profile of the sample 710 based oncross-correlations among the plurality of images.

FIG. 8 shows a schematic diagram of an imaging system 800 according tosome embodiments. The imaging system 800 is similar to the imagingsystem 700 illustrated in FIG. 7, but may further include an objectivelens 860 and a tube lens 870 positioned between the mask 750 and theimage sensor 740, as in a conventional microscope.

According to some embodiments, the reconstruction process may includethe following steps.

At S301, initialize the complex object O(x,y) (e.g., the sample) and thediffuser pattern P(x,y).

At S302, estimate the j^(th) translated position of the sample(x_(j),y_(j)) based on image cross-correlation, or other trackingalgorithms such as mutual information optimization and the like.

At S303, according to the imaging model, O(x−x_(j),y−y_(j)) ispropagated for distance ‘d₁’ to the diffuser planeO _(d1)(x−x _(j) ,y−y _(j))=O(x−x _(j) ,y−y _(j))*PSF _(free)(d ₁).

At S304, generate the j^(th) complex image's exit wave ψ_(j)(x,y) at theimage sensor plane based on the translated diffuser position(x_(j),y_(j)), O(x,y), and P(x,y):ψ_(j)(x,y)=(O _(d1)(x−x _(j) ,y−y _(j))·P(x,y))*PSF_(free)(d)=φ_(j)(x,y)*PSF _(free)(d),where PSF_(free)(d) is the point spread function (PSF) for free-spacepropagation over a distance d, and ‘*’ stands for convolution operation,and φ_(j)(x,y)=O_(d1)(x−x_(j),y−y_(j))·P(x,y).

At S305, at the image sensor plane, use the following equation to updatethe exit wave ψ_(j)(x,y) based on the captured intensity imageI_(j)(x,y):

${\psi_{j}^{\prime}\left( {x,y} \right)} = {{\psi_{j}\left( {x,y} \right)}{\left( \frac{\sqrt{{I_{j}\left( {x,y} \right)}_{\uparrow M}}}{\sqrt{{{\psi_{j}\left( {x,y} \right)}}^{2}*{{ones}\left( {M,M} \right)}_{\downarrow M \uparrow M}}} \right).}}$

In the above equation, the image sizes of ψ_(j)(x,y) and I_(j)(x,y) aredifferent. If I_(j) has a size of 100 by 100 pixels, ψ_(j) will have 300by 300 pixels, with an up-sampling factor M=3. The term‘I_(j)(x,y)_(↑M)’ represents the nearest-neighbor up-sampling of thecaptured image I_(j). In the denominator of the above equation, theterm|ψ_(j)(x,y)|² first convolutes with an average filter (M by Mall-one matrix ones(M,M)). It will be then down-sampling by M-timesfollowed by nearest-neighbor up-sampling of M-times. In someembodiments, other up-sampling factor (e.g., M=4, 5, 6, . . . ) may beused.

At S306, propagate the updated ψ′_(j)(x,y) to the object plane and getφ′_(j)(x_(j),y_(j)). Update the object O_(d1)(x,y) and P(x,y):

${{O_{d\; 1}\left( {{x - x_{j}},{y - y_{j}}} \right)} = {{O_{d\; 1}\left( {{x - x_{j}},{y - y_{j}}} \right)} + \frac{{{conj}\left( {P\left( {x,y} \right)} \right)}\left( {\varphi_{j}^{\prime} - \varphi_{j}} \right)}{{\left( {1 - \alpha_{obj}} \right){{P\left( {x,y} \right)}}^{2}} + {\alpha_{obj}{{P\left( {x,y} \right)}}_{\max}^{2}}}}},{{P\left( {x,y} \right)} = {{P\left( {x,y} \right)} + \frac{{{conj}\left( {O_{d\; 1}\left( {{x - x_{j}},{y - y_{j}}} \right)} \right)}\left( {\varphi_{j}^{\prime} - \varphi_{j}} \right)}{{\left( {1 - \alpha_{P}} \right){{O_{d\; 1}\left( {{x - x_{j}},{y - y_{j}}} \right)}}^{2}} + {\alpha_{P}{{O_{d\; 1}\left( {{x - x_{j}},{y - y_{j}}} \right)}}_{\max}^{2}}}}},$where ‘conj’ denotes conjugate, and α_(obj) and α_(P) are algorithm.

At S307, j=j+1 and repeat steps S302-S306.

At S308, repeat steps S302-S30 until the solution converges.

At S309, propagates the recovered O_(d1)(x,y) to the object plane.

It should be appreciated that the specific steps S301-S309 discussedabove provide a particular reconstruction process according to someembodiments. Other sequences of steps may also be performed according toalternative embodiments. For example, alternative embodiments of thepresent invention may perform the steps outlined above in a differentorder. Moreover, the individual steps S301-S309 may include multiplesub-steps that may be performed in various sequences as appropriate tothe individual step. Furthermore, additional steps may be added orremoved depending on the particular applications. One of ordinary skillin the art would recognize many variations, modifications, andalternatives.

Imaging via Pattern Modulation by Scanning a Sample

FIG. 9 shows a schematic diagram of an imaging system 900 according tosome embodiments. The imaging system 900 includes an image sensor 940. Asample 910 to be imaged may be placed above the image sensor 940. Forexample, the sample 910 may be held by a sample mount (not shown in FIG.9).

The imaging system 900 further includes a light source 920. The lightsource 920 may comprise a laser or a light-emitting diode (LED), and isconfigured to emit a coherent or partially coherent light beam to beincident on the sample 910. The light beam may be collimated, partiallycollimated, or uncollimated. The imaging system 900 may include a mirror930 positioned substantially at a 45 degree angle with respect to thepath of the light beam emitted by the light source 920, so as to foldthe light beam for a more compact configuration. The mirror 930 isoptional.

The imaging system 900 further includes a mask 950 positioned downstreamfrom the sample 910 along an optical axis 902 of the imaging system 900,and above the image sensor 940. The mask 950 may include an unknownpattern formed thereon. Thus, as the light beam is transmitted throughthe sample 910 and the mask 950, a diffused image may be formed at theimage sensor 940. In some embodiments, the mask 950 may include an area952 that is free of the pattern. Thus, the image sensor 940 may detectan image of a feature on the sample 910. By tracking the movement of thefeature, the movement of the sample 910 may be tracked. The detectedpositional shift of the sample is used to recover the sample and/or themask profile in the reconstruction process.

The imaging system 900 further includes a translation mechanism (notshown in FIG. 9) coupled to the sample mount and configured to scan thesample 910 to a plurality of sample positions in a plane (e.g., the X-Yplane) substantially perpendicular to the optical axis 902 of theimaging system 900. According to various embodiments, the sample 910 maybe translated in one dimension (e.g., in the X direction or the Ydirection), or in two dimensions (e.g., in both the X and the Ydirections).

As the sample 910 is scanned, the diffused image formed at the imagesensor 940 may shift accordingly. The image sensor 940 is configured tocapture a plurality of images as the sample 910 is scanned to theplurality of sample positions. Each respective image corresponds to arespective sample position. The plurality of images may be processed bya processor (not shown in FIG. 9) to produce a complex profile of thesample 910 based on cross-correlations among the plurality of images.

According to some embodiments, the reconstruction process may includesteps similar to steps S301-S309 as discussed above.

Imaging via Translated Phase Modulation Using a Height-Varying PhaseMask

FIG. 10A shows a schematic diagram of an imaging system 1010 accordingto some embodiments. The imaging system 1010 is similar to the imagingsystem 900 illustrated in FIG. 9A, except that the mask 950 is replacedby a phase mask 1012. The phase mask 1012 may comprise a transparentplate with varying thicknesses across the lateral plane (e.g., the X-Yplane) of the phase mask 1012. In the embodiment illustrated in FIG.10A, the phase mask 1012 may be a wedge-shaped prism, with its thicknessvarying continuously (e.g., linearly) along the X-axis (or the Y-axis).FIG. 10B shows an alternative embodiment, in which a phase mask 1022 hasa step-like cross section with its thickness varying discretely alongthe X-axis (or the Y-axis). According to various embodiments, thethickness of the phase mask 1012 may vary in various ways. For example,the thickness may vary in a non-linear fashion from one side to theother, or may vary in a non-monotonically from one side to the other. Insome embodiments, the thickness of the phase mask 1012 may vary in arandom fashion.

According to some embodiments, the reconstruction process may includesteps similar to steps S301-S309 as discussed above.

Imaging via Translated Phase Modulation With a Tilted Image Sensor

FIG. 11 shows a schematic diagram of an imaging system 1100 according tosome embodiments. The imaging system 1100 is similar to the imagingsystem 900 illustrated in FIG. 9A, but here, the mask 950 is omitted andthe image sensor 940 is tilted with respect to the optical axis 902 ofthe imaging system 1100. The air gap between the sample 910 and theimage sensor 940 may serve as a height-varying phase mask.

According to some embodiments, the reconstruction process may includesteps similar to steps S301-S309 as discussed above.

Imaging via Wavelength-Encoded Mask Modulation

FIG. 12 shows a schematic diagram of an imaging system 1200 according tosome embodiments. The imaging system 1200 includes an image sensor 1240.A sample 1210 to be imaged may be placed above the image sensor 1240.For example, the sample 1210 may be held by a sample mount (not shown inFIG. 12).

The imaging system 1200 further includes a light source 1220. The lightsource 1220 is configured to emit a light beam 1270 of multiplewavelengths. The light beam 1270 may be collimated, partiallycollimated, or uncollimated. In some embodiments, the light source 1220may comprise multiple light-emitting elements (e.g., 3, 5, or up to 20laser diodes) configured to emit light in different wavelengths.Alternatively, the light source 1220 may comprise a broadband lightsource, for example, a broadband light-emitting diode (LED). The imagingsystem 1200 may include a mirror 1230 positioned substantially at a 45degree angle with respect to the path of the light beam 1270 emitted bythe light source 1220, so as to fold the light beam 1270 for a morecompact configuration. The mirror 1230 is optional.

The imaging system 1200 further includes a light dispersing element 1260configured to receive and disperse the light beam 1270 into a pluralityof sub light beams 1272 a, 1272 b, and 1272 c, each sub light beam 1272a, 1272 b, or 1272 c corresponding to a respective wavelength. The lightdispersing element 1260 may comprise, for example, a prism, an opticaldiffraction grating, or the like. Although only three sub light beamsare illustrated in FIG. 12, there can be more or fewer than three sublight beams according to various embodiments. The plurality of sub lightbeams 1272 a, 1272 b, and 1272 c may be incident on the sample atdifferent angles of incidence.

The imaging system 1200 further includes a mask 1250 positioneddownstream from the sample 1210 along an optical axis 1202 of theimaging system 1200, and above the image sensor 1240. The mask 1250 mayinclude an unknown pattern formed thereon. Thus, as the plurality of sublight beams 1272 a, 1272 b, and 1272 c is transmitted through the sample1210 and the mask 1250, a diffused image may be formed at the imagesensor 1240. The diffused image may be a superposition of a plurality ofsub-images corresponding to the different wavelengths of the pluralityof sub light beams 1272 a, 1272 b, and 1272 c. Since the plurality ofsub light beams 1272 a, 1272 b, and 1272 c is incident on the mask 1250at different angles of incidence, the light modulation produced by themask 1250 may be wavelength-dependent. The wavelength-dependent featureof the light modulation may be used to recover the profiles of thesample 1210 at different wavelengths in the phase retrieval process.

In some embodiments, the mask 950 may include an area 952 that is freeof the pattern. Thus, the image sensor 940 may detect an image of afeature on the sample 910. By tracking the movement of the feature, themovement of the sample 910 may be tracked. The detected positional shiftof the sample is used to recover the sample and/or the mask profile inthe reconstruction process.

The imaging system 1200 further includes a translation mechanism (notshown in FIG. 12) coupled to the sample mount and configured to scan thesample 1210 to a plurality of sample positions in a plane (e.g., the X-Yplane) substantially perpendicular to the optical axis 1202 of theimaging system 1200. According to various embodiments, the sample 1210may be translated in one dimension (e.g., in the X direction or the Ydirection), or in two dimensions (e.g., in both the X and the Ydirections).

As the sample 1210 is scanned, the diffused image (e.g., a superpositionof a plurality of sub-images corresponding to the different wavelengths)formed at the image sensor 1240 may shift accordingly. The image sensor1240 is configured to capture a plurality of images as the sample 1210is scanned to the plurality of sample positions. Each respective imagecorresponds to a respective sample position. The plurality of images maybe processed by a processor (not shown in FIG. 12) to recover complexprofiles of the sample 1210 at different wavelengths based oncross-correlations among the plurality of images.

According to some embodiments, the reconstruction process may includethe following steps.

At S401, initialize multiple object (e.g., sample) estimates O_(t)(x,y)and the diffuser pattern or the modulation mask pattern P_(t)(x,y),where ‘t=1,2 . . . T’. T represents the number of wavelengths used forillumination.

At S402, estimate the translated position of the translated sampleposition (x_(i),y_(i)) based on cross-correlation or mutual informationof the captured images or other tracking algorithms.

At S403, according to the imaging model, O_(t)(x,y) is propagated ‘d₁’to the modulate plane based on translated position (x_(i),y_(i)), toobtain:O _(t,d) ₁ (x−x _(i) ,y−y _(i))=O _(t)(x−x _(i) ,y−y _(i))*PSF _(free)(d₁).Then generate the corresponding target image I_(t,i)(x,y) at the imagesensor plane as follows:

I_(t, i)(x, y) = O_(t, d₁)(x − x_(i), y − y_(i)) ⋅ P_(t)(x, y) * PSF_(free)(d₂)_( ↓ M)² = φ_(t, i)(x, y) * PSF_(free)(d₂)_( ↓ M)² = ψ_(t, i)(x, y)_( ↓ M)²,where ‘·’ stands for point-wise multiplication, and ‘*’ denotes theconvolution operation. ‘d₁’ is the distance between the object and thediffuser, and ‘d₂’ is the distance between the diffuser and the imagesensor. PSF_(free)(d) is used to model the point spread function (PSF)for free-space propagation over distance ‘d’. ‘↓M’ in the above equationrepresents the down-sampling process.

At S404, sum I_(t,i)(x,y) up to generate the incoherent mixture:I _(incoherent,i)(x,y)=Σ_(t=1) ^(T) I _(t,i)(x,y).

At S405, update ψ_(t,i)(x,y) using the ratio between the actualmeasurement I_(m,i)(x,y) and I_(incoherent)(x_(i),y_(i)) and keep thephase unchanged:

${\psi_{t,i}^{\prime}\left( {x,y} \right)} = {{\psi_{t,i}\left( {x,y} \right)}\left( \frac{\sqrt{{I_{m,i}\left( {x,y} \right)}_{\uparrow M}}}{\sqrt{{I_{{incoherent},i}\left( {x,y} \right)}*{{ones}\left( {M,M} \right)}_{\downarrow M \uparrow M}}} \right)}$The term I_(m,i)(x,y)_(↑M) represents the nearest-neighbor up-samplingof the captured image I_(m,i)(x,y). In the denominator of equation, theterm I_(incoherent,i)(x,y) first convolutes with an average filter (M byM all-ones matrix). It will be then down-sampled by M-times followed byM-times nearest-neighbor up-sampling. In some embodiments, otherup-sampling factor (e.g., M=4, 5, 6 . . . ) may be used.

At S406, back propagate ψ′_(t,i)(x,y) to the modulate plane:ψ′_(t,i)(x,y)=ψ′_(t,i)(x,y)*PSF _(free)(−d ₂).

At S407, update O_(t,d) ₁ (x−x_(i),y−y_(i)) and modulation mask patternP_(t)(x,y):

${O_{t,d_{1}}^{update}\left( {{x - x_{i}},{y - y_{i}}} \right)} = {{O_{t,d_{1}}\left( {{x - x_{i}},{y - y_{i}}} \right)} + \frac{{{conj}\left( {P_{t}\left( {x,y} \right)} \right)} \cdot \left\{ {{\varphi_{t,i}^{\prime}\left( {x,y} \right)} - {\varphi_{t,i}\left( {x,y} \right)}} \right\}}{{\left( {1 - \alpha_{obj}} \right){{O_{t,d_{1}}\left( {{x - x_{i}},{y - y_{i}}} \right)}}^{2}} + {\alpha_{obj}{{O_{t,d_{1}}\left( {{x - x_{i}},{y - y_{i}}} \right)}}_{\max}^{2}}}}$${P_{t}^{update}\left( {x,y} \right)} = {{P_{t}\left( {x,y} \right)} + {\frac{{{conj}\left( {O_{t,d_{1}}\left( {{x - x_{i}},{y - y_{i}}} \right)} \right)} \cdot \left\{ {{\varphi_{t,i}^{\prime}\left( {x,y} \right)} - {\varphi_{t,i}\left( {x,y} \right)}} \right\}}{{\left( {1 - \alpha_{p}} \right){{P_{t}\left( {x,y} \right)}}^{2}} + {\alpha_{p}{{P_{t}\left( {x,y} \right)}}_{\max}^{2}}}.}}$

At S408, update the shifted object O_(t)(x−x_(i),y−y_(i)) using:O _(t) ^(update)(x−x _(i) ,y−y _(i))=O _(t,d) ₁ ^(update)(x−x _(i) ,y−y_(i))*PSF _(free)(−d ₁).

At S409, j=j+1 and repeat steps S402-S408.

At S410, repeat steps S402-S409 until the solution converges.

At S411, propagate the recovered O_(t) ^(update)(x,y) to the objectplane.

It should be appreciated that the specific steps S401-S411 discussedabove provide a particular reconstruction process according to someembodiments. Other sequences of steps may also be performed according toalternative embodiments. For example, alternative embodiments of thepresent invention may perform the steps outlined above in a differentorder. Moreover, the individual steps S401-S411 may include multiplesub-steps that may be performed in various sequences as appropriate tothe individual step. Furthermore, additional steps may be added orremoved depending on the particular applications. One of ordinary skillin the art would recognize many variations, modifications, andalternatives.

An advantage of the imaging systems illustrated in FIGS. 5-9, 10A-10B,11, and 12 may be that diffuser modulation is performed at the detectionpath. Different from illumination-based approaches, the recovered imagedepends only on how the complex wavefront exits the sample. Therefore,the sample thickness becomes irrelevant during reconstruction. Afterrecovery, the complex wavefront may be propagated to any position alongthe optical axis.

This concept was validated using a thick potato sample. FIG. 13 showsthe recovered images. The inset (a) shows the recovered amplitude imageof the exit wavefront from the sample. The insets (b1), (b2), and (b3)show the recovered amplitude image of the sample after digitallypropagating to z=620 μm, z=650 μm, and z=685 μm, respectively. The cellwalls are in focus in the inset (b1), and the organelles are in focus inthe insets (b2) and (b3).

Three-Dimensional Tomographic Imaging

According to some embodiments, the imaging systems described above, thelight source may be replaced by a light source array, such as an LEDarray. Different light sources in the light source array may illuminatethe sample at different angles of incidence. A plurality of complexprofiles of the sample may be recovered, each respective profilecorresponding to a respective light source. A three-dimensionaltomographic image of the sample may be reconstructed from the pluralityof complex profiles of the sample.

The imaging systems discussed above according to embodiments of thepresent invention may afford numerous advantages. For example, it is notnecessary to know the position of the speckle pattern or the maskmodulation. Therefore, the image acquisition process can be free-run.That is, any scanning motion (e.g., the scanning of the mirror 130 shownin FIG. 1, the scanning of the mask 550 shown in FIGS. 5 and 6, thescanning of the transparent plates 760 and 770 shown in FIGS. 7 and 8,the scanning of the sample 910 shown in FIGS. 9, 10A-10B, and 11, andthe scanning of the sample 1210 shown in FIG. 12) can be run “blindly”without precise control, such as synchronization, triggering, timing, orthe like. In addition, the use of low-cost galvo scanner in the imagingsystem shown in FIG. 1 may be advantageous compared to conventionalmulti-height implementations. The mechanical scanning time may benegligible according to embodiments of the present invention.Furthermore, the imaging systems may provide true quantitative contrastof a complex object. The imaging systems may provide solutions foraddressing point-of-care and telemedicine related challenges.

It is also understood that the examples and embodiments described hereinare for illustrative purposes only and that various modifications orchanges in light thereof will be suggested to persons skilled in the artand are to be included within the spirit and purview of this applicationand scope of the appended claims.

What is claimed is:
 1. An imaging system comprising: a sample mount forholding a sample to be imaged; a light source configured to emit a lightbeam to be incident on the sample; a translation mechanism coupled tothe sample mount and configured to scan the sample to a plurality ofsample positions in a plane substantially perpendicular to an opticalaxis of the imaging system; a mask positioned downstream from the samplealong the optical axis; an image sensor positioned downstream from themask along the optical axis, the image sensor configured to acquire aplurality of images as the sample is translated to the plurality ofsample positions, each respective image corresponding to a respectivesample position; and a processor configured to process the plurality ofimages to recover a complex profile of the sample based on positionalshifts extracted from the plurality of images.
 2. The imaging system ofclaim 1, wherein the positional shifts are extracted from the pluralityof images using cross-correlation or mutual information among theplurality of images.
 3. The imaging system of claim 1, wherein thesample is scanned in one dimension or two dimensions.
 4. The imagingsystem of claim 1, wherein the complex profile of the sample comprisesan intensity image and a phase image of the sample.
 5. The imagingsystem of claim 1, wherein the light beam is coherent or partiallycoherent.
 6. The imaging system of claim 1, wherein the mask ispositioned adjacent a top surface of the image sensor.
 7. The imagingsystem of claim 1, wherein the mask comprises a diffuse pattern formedthereon.
 8. The imaging system of claim 7, wherein a portion of the maskis free of the diffuse pattern.
 9. The imaging system of claim 7,wherein the processor is further configured to process the plurality ofimages to recover an image of the diffuse pattern.
 10. The imagingsystem of claim 1, wherein the mask comprises a phase mask.
 11. Theimaging system of claim 10, wherein the phase mask comprises a slab of atransparent material with varying thicknesses across a lateral planeperpendicular to the optical axis of the imaging system.
 12. The imagingsystem of claim 11, wherein the phase mask comprises a wedge-shapedprism.
 13. The imaging system of claim 11, wherein the phase mask ischaracterized by a multi-step cross section.